[https://nvbugs/6490049][test] Unwaive GB300 Kimi disagg e2e by chienchunhung · Pull Request #16920 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-07-27T23:49:39Z

Summary

remove only the waiver for the GB300 Kimi-K2.5 Thinking FP4 concurrency-4096 disaggregated e2e perf-sanity case tracked by NVBUG 6490049
preserve the Python cache transceiver default kv_transfer_timeout_ms=60000; this PR has no net timeout override
keep the already-unwaived gen_only case unchanged; that case is tracked separately by NVBUG 6487038

Why

main still skips this exact e2e case under NVBUG 6490049. The corresponding gen_only case was already unwaived on main by #16717, so it is not part of this PR's waiver diff.

This PR was originally opened during the NVBUG 6487038 investigation. After rebasing onto the changes from #16717 and #16832, the remaining net waiver change is specifically the e2e case owned by NVBUG 6490049. The PR is therefore retargeted to that bug.

Stack

stacked above [None][fix] Align perf launcher with pytest shard #17134 so the multi-node launcher and pytest select the same duration-balanced shard
stacked above [None][fix] Bound GEN log sentinel wait #17140, which prevents the GEN-log completion sentinel from blocking e2e, where the GEN-only device-step-time metric is not collected

The verification below was collected on the earlier patch-equivalent combined stack at commit e0bfe69c484e2986528bca0ec6dfab1632d263c2. The current restacked head is e4f6185c312a30b2e5dad03c0efce1e4c30263f5; git range-diff confirms all three #16920-specific commits are unchanged.

Verification status

COMPLETE: every test unwaived by this PR has an exact targeted pass (1/1).

Scope	Exact test	Net waiver change in this PR	Evidence	Result	KV-transfer timeout
NVBUG 6490049	`perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL]`	Yes	pipeline #51208, CI report, completion comment	PASSED	60,000 ms
NVBUG 6487038 (supplemental)	`perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL]`	No; already unwaived by #16717	pipeline #51223, CI report, completion comment	PASSED	60,000 ms

The e2e run reported 1 passed, 2 skipped, 4153 deselected; its test call duration was 1941.99s and pytest total was 1951.11s. The GEN-only run also completed pytest and Slurm teardown successfully.

The YAML has no explicit GEN or CTX kv_transfer_timeout_ms override, so both validations use the Python transceiver's existing 60-second default.

QA-weekly scope

This PR does not unwaive a QA-weekly test. The weekly lists already contain the Kimi concurrency-4096 NIXL and UCX disagg-e2e entries without SKIP markers. They are distinct from the post-merge disagg_upload-e2e node unwaived here.

The QA local launcher receives one exact --test-list node ID and writes a one-entry test list. It therefore has no pytest-split shard-selection mismatch to mirror from #17134's CI-launcher fix.

Scope

The PR removes one e2e waiver. It does not change the global KV-transfer timeout, add a GB300 timeout override, modify the inherited GEN-only waiver state, or add/remove a QA-weekly waiver.

chienchunhung · 2026-07-27T23:50:08Z

/bot run --disable-fail-fast --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4"

tensorrt-cicd · 2026-07-27T23:56:45Z

PR_Github #62031 [ run ] triggered by Bot. Commit: 880e087 Link to invocation

chienchunhung · 2026-07-27T23:57:51Z

/bot run --disable-fail-fast --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4"

coderabbitai · 2026-07-28T00:02:08Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

Adds duration-balanced pytest-split performance test selection. The submit flow reuses the selected test for configuration and output naming. Generation-log synchronization now has a dedicated timeout and applies metric collection only to gen_only runs. One performance waiver is removed.

Changes

Pytest-split performance selection

Layer / File(s)	Summary
Duration-balanced shard selection `jenkins/scripts/perf/submit.py`	Parses pytest options, loads duration data, reconstructs least-duration groups, and validates the selected shard.
Selected test integration `jenkins/scripts/perf/submit.py`	Reuses the selected test line for configuration parsing and output naming. Updates split-group help text.
Selection validation coverage `tests/unittest/scripts/test_perf_submit.py`	Verifies least-duration shard selection and rejects conflicting split-group values.

Generation-log synchronization

Layer / File(s)	Summary
Generation-only metric collection `tests/integration/defs/perf/test_perf_sanity.py`	Adds a dedicated sentinel timeout and limits log snapshots, waits, and device-step-time parsing to `gen_only` runs.

Performance waiver management

Layer / File(s)	Summary
Remove performance test waiver `tests/integration/test_lists/waives.txt`	Deletes the waiver for the specified disaggregated upload end-to-end performance test.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SubmitScript
  participant PytestCommand
  participant DurationData
  participant ShardSelector
  SubmitScript->>PytestCommand: Parse split options
  SubmitScript->>DurationData: Load test durations
  DurationData-->>ShardSelector: Provide test durations
  PytestCommand-->>ShardSelector: Provide split and group values
  ShardSelector-->>SubmitScript: Return one selected test

Possibly related PRs

NVIDIA/TensorRT-LLM#15585: Modifies the same waiver list.
NVIDIA/TensorRT-LLM#16725: Modifies the performance submit script flow.
NVIDIA/TensorRT-LLM#17128: Removes a waiver entry from the same list.

Suggested reviewers: brnguyen2, bowenfu, mzweilz, chenfeiz0326, qijune

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the NVBugs issue and the primary change: unwaiving the GB300 Kimi disaggregated end-to-end performance test.
Description check	✅ Passed	The description explains the issue, scope, stacked changes, test coverage, verification results, and preserved timeout behavior.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

chienchunhung · 2026-07-28T00:05:47Z

/bot help

github-actions · 2026-07-28T00:05:59Z

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". The patterns "*", "*Post-Merge*", and "*PerfSanity*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label. Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline. Requires the ci: post-merge approved PR label applied by an active member of NVIDIA/trt-llm-ci-approvers. The approval label remains in place when new commits are pushed.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge". The patterns "*", "*Post-Merge*", and "*PerfSanity*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label.

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

chienchunhung · 2026-07-28T00:06:31Z

/bot kill

tensorrt-cicd · 2026-07-28T00:31:23Z

PR_Github #62033 [ kill ] triggered by Bot. Commit: dd8d4db Link to invocation

tensorrt-cicd · 2026-07-28T00:35:07Z

PR_Github #62031 [ run ] completed with state ABORTED. Commit: 880e087

Link to invocation

tensorrt-cicd · 2026-07-28T00:35:16Z

PR_Github #62033 [ kill ] completed with state SUCCESS. Commit: dd8d4db
Successfully killed previous jobs for commit dd8d4db

Link to invocation

BowenFu

Test-only: perf-sanity YAML gains kv_transfer_timeout_ms: 600000 (matching the equivalent GB200 Kimi config) and the two matching waivers come out. No product source, no default change — kv_transfer_timeout_ms is an existing per-config knob and the global 60s default is untouched. The single YAML serves both the e2e and gen_only variants, so both unwaived cases are covered.

Two notes, neither blocking:

The targeted GB300 stages were aborted rather than green, so the e2e case (waived under the broader nvbugs/6490049) isn't yet proven to pass for this reason alone.
#16921 removes the same disagg_upload-e2e-gb300_kimi-...con4096... waiver line, so whichever lands second will need a rebase.

chienchunhung · 2026-07-28T01:18:24Z

/bot run --disable-fail-fast --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4"

chienchunhung · 2026-08-03T17:44:49Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-03T17:51:35Z

PR_Github #63513 [ run ] triggered by Bot. Commit: 83ab4dc Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-08-03T18:01:55Z

/bot run --disable-fail-fast

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-08-03T18:07:44Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-03T18:09:06Z

PR_Github #63518 [ run ] triggered by Bot. Commit: cbd1a5e Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

…meout Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

tensorrt-cicd · 2026-08-03T18:12:57Z

PR_Github #63513 [ run ] completed with state ABORTED. Commit: 83ab4dc

Link to invocation

chienchunhung · 2026-08-03T18:13:49Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-03T18:14:25Z

PR_Github #63520 [ run ] triggered by Bot. Commit: e4f6185 Link to invocation

tensorrt-cicd · 2026-08-03T18:18:08Z

PR_Github #63518 [ run ] completed with state ABORTED. Commit: cbd1a5e

Link to invocation

tensorrt-cicd · 2026-08-03T18:21:50Z

PR_Github #63527 [ run ] triggered by Bot. Commit: e4f6185 Link to invocation

tensorrt-cicd · 2026-08-03T18:25:51Z

PR_Github #63520 [ run ] completed with state ABORTED. Commit: e4f6185

Link to invocation

tensorrt-cicd · 2026-08-03T22:08:45Z

PR_Github #63527 [ run ] completed with state SUCCESS. Commit: e4f6185
/LLM/main/L0_MergeRequest_PR pipeline #51493 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv

I'm not familiar with the perf sanity code paths, I'll defer to other reviewers in the qa review group.

github-actions Bot assigned chienchunhung Jul 27, 2026

chienchunhung requested review from Shixiaowei02 and chenfeiz0326 July 27, 2026 23:58

chienchunhung marked this pull request as ready for review July 27, 2026 23:59

chienchunhung requested review from a team as code owners July 27, 2026 23:59

chienchunhung requested review from BowenFu, brnguyen2, dc3671, fredricz-20070104, mlefeb01, mzweilz, pcastonguay and yufeiwu-nv July 27, 2026 23:59

coderabbitai Bot mentioned this pull request Jul 28, 2026

[https://nvbugs/6481034][test] Trace Kimi gen-only KV transfer timeout #16918

Closed

BowenFu approved these changes Jul 28, 2026

View reviewed changes

chienchunhung added 3 commits August 3, 2026 10:54

[NVBUG-6481034][fix] Align perf launcher with pytest shard

d4a5122

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG-6481034][fix] Address shard selection review feedback

9b6d78c

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG-6481034][test] Guard pytest-split compatibility

63f2737

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/nvbug-6487038-kv-transfer-timeout branch from 83ab4dc to 47846e9 Compare August 3, 2026 18:00

[NVBUG-6481034][fix] Address selector contract review

9429527

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/nvbug-6487038-kv-transfer-timeout branch from 47846e9 to cbd1a5e Compare August 3, 2026 18:05

chienchunhung added 9 commits August 3, 2026 11:10

[NVBUG-6481034][refactor] Use one shard selection path

79f24c6

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[https://nvbugs/6487038][fix] Bound GEN log sentinel wait

3962553

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG-6487038][fix] Address perf review feedback

d3e54b6

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG-6487038][fix] Preserve GEN log metric fallback

a469ce2

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG-6487038][test] Preserve pytest-split contract

e8ab8ad

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG-6487038][test] Type sentinel fallback test

1cefe9c

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[https://nvbugs/6487038][fix] extend GB300 Kimi KV transfer timeout

ce69d75

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[https://nvbugs/6487038][test] unwaive GB300 Kimi KV timeout cases

dd8607b

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[https://nvbugs/6487038][test] validate GB300 Kimi with default KV ti…

e4f6185

…meout Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/nvbug-6487038-kv-transfer-timeout branch from cbd1a5e to e4f6185 Compare August 3, 2026 18:13

dpitman-nvda approved these changes Aug 3, 2026

View reviewed changes

tburt-nv reviewed Aug 4, 2026

View reviewed changes

Uh oh!

Conversation

chienchunhung commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Stack

Verification status

QA-weekly scope

Scope

Uh oh!

chienchunhung commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

chienchunhung commented Jul 27, 2026

Uh oh!

coderabbitai Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

chienchunhung commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

chienchunhung commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

BowenFu left a comment

Choose a reason for hiding this comment

Uh oh!

chienchunhung commented Jul 28, 2026

Uh oh!

chienchunhung commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

chienchunhung commented Aug 3, 2026

Uh oh!

chienchunhung commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

chienchunhung commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tburt-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

chienchunhung commented Jul 27, 2026 •

edited

Loading

coderabbitai Bot commented Jul 28, 2026 •

edited

Loading